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Q\ \ Abstract 
a^ ■ 

■ We investigate the indications of flavor oscillations that come from the 

1-^ ■ anomalous flavor composition of the atmospheric neutrino flux observed in 

lO . some underground experiments. We study the information coming from the 

^ \ neutrino-induced ^-like and e-like events both in the sub-GeV energy range 

^ \ (Kamiokande, 1MB, Frejus, and NUSEX experiments) and in the multi-GeV 

' energy range (Kamiokande experiment). First we analyze all the data in the 

' limits of pure <^ and ^ Vf. oscillations. We obtain that <-> Vf, 

. oscillations provide a better fit, in particular to the multi-GeV data. Then we 

\^ \ perform a three-flavor analysis in the hypothesis of dominance of one neutrino 

0^ \ square mass difference, m^, implying that the neutrino mixing is parametrized 

^ ■ by two angles, (V', </>) G [0, 7r/2]. We explore the space (m^, V', 4>) exhaus- 

^p^' tively, and find the regions favored by the oscillation hypothesis. The results 
are displayed in a form suited to the comparison with other fiavor oscillation 

r~| ! searches at accelerator, reactor, and solar v experiments. In the analysis, we 

^ \ pay particular attention to the earth matter effects, to the correlation of the 

' uncertainties, and to the symmetry properties of the oscillation probability. 

H : PACS number(s): 14.60.Pq, 95.85.Ry, 13.15.-hg 
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I. INTRODUCTION 



The indication for an anomalous muon and electron flavor composition of the observed 
atmospheric neutrino flux represents a still unsolved puzzle (for recent reviews, see [0]). A 
possible explanation could be provided by neutrino flavor oscillations. In this paper we adopt 
such a viewpoint and explore systematically its consequences in a three-flavor framework 
with one dominant neutrino square mass difference. 

We perform a comprehensive and accurate analysis of the experimental information com- 
ing from the Kamiokande f^-Q, 1MB |^,||, Frejus and NUSEX atmospheric neutrino 
experiments. We do not characterize the atmospheric neutrino anomaly with the popular 
double flavor ratio R^/e = (/i/e)data/(At/e)theory which, as discussed in |jTO|, is affected by 



non-Gaussian uncertainties. Instead we separate the //-like and e-like event rates, whose 



errors are normally distributed |T^. We compare the data with the theoretical expectations 
in the presence of two-flavor and three-flavor oscillations, including the earth matter effects 
ri|,IT2|. We place significant bounds in the oscillation parameter space, which can provide 



useful guidelines for model building, for the expectations at the running SuperKamiokande 
experiment |TB[, and for the discovery potential at future long-baseline neutrino oscillation 



experiments |TJ] . We also highlight the interplay between the results of this work and those 
obtained — within the same theoretical framework — from the analysis of flavor oscillation 
searches in accelerator and reactor ||T5[| or solar [|I6| neutrino experiments. We include a 
larger data set and use a more refined statistical approach than previous three-flavor analy- 
ses of the atmospheric neutrino anomaly performed by other authors [p!^^-p7| and by ourselves 
11,111111 • 



The paper has the following structure. In Sec. II we introduce the experimental ingredi- 
ents of the analysis and describe their treatment. In Sec. Ill we briefly recall the properties of 
three-flavor oscillations in the hypothesis of one dominant square mass difference. In Sec. IV 
we perform the data analysis in the subcases of two-flavor v-r and ^ Ve oscillations. 
In Sec. V we present the main results of our analysis of three-flavor atmospheric neutrino 
oscillations and discuss their relation with the indications coming from other (laboratory 
and solar) neutrino oscillation experiments. In Sec. VI we summarize our work and draw 
our conclusions. In Appendices A, B, and C, we discuss respectively our treatment of the 
Kamiokande multi-GeV neutrino data, the correlation of the theoretical uncertainties, and 
the symmetry properties of the neutrino oscillation probability. 



II. EXPERIMENTAL INGREDIENTS AND THEIR ANALYSIS 

In this section we introduce the experimental data and discuss briefly some technical 
aspects of their analysis. 



A. Experimental data 

We analyze the largest set of data on the electron and muon composition of the atmo- 
spheric neutrino flux, which can be considered both homogeneous and statistically consistent. 
In particular, we include the neutrino-induced e-like and /z-like event rates measured by the 
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four experiments Kamiokande P,^, 1MB Frejus and NUSEX at low energies 
(so-called sub-GeV data), and by the Kamiokande experiment at higher energies in five 
zenith-angle sectors (so-called multi-GeV data).Q Concerning the multi-GeV data, we 
will consider either the information coming from the binned angular distribution of events 
("binned multi-GeV"), or the reduced information coming from the angle-integrated number 
of events ("unbinned multi-GeV"). 

As it is well known, the Frejus and NUSEX measurements do not confirm the Kamiokande 
and 1MB observation of an anomalous muon-electron fiavor composition of the atmospheric 
events However, all the data are statistically compatible within ~ 2cr or better (see 

Fig. 3 in [^) and their combination in a global analysis is reasonable. 

We do not include the higher-energy data pertaining only to the muon fiavor content. 



the so-called upward-going muons (see, e.g., pll). This data set has specific experimental 



characteristics and deserves a separate theoretical analysis (in progress). However, we will 
briefiy comment on upward-going muons in due course.FI We also do not include the data from 



the Soudan 2 experiment ^3[) because a detailed, official analysis has not been published 
yet by the experimental collaboration. 

We emphasize that, according to the discussion in |jlO[, we prefer to separate the electron 
and muon fiavor information, instead of using the popular ratio R^/e = (/i/e)data/ (/^/e)theory 
The ratio R^/e allows a large cancellation of the theoretical errors |^0[, but its probability 
distribution is highly non-Gaussian.^ We have shown that one can use only normally 
distributed variables provided that, when the and e experimental and theoretical rates are 
separated, the correlations of their uncertainties are included. 

In the comparison of the data with theoretical predictions, we use a statistic including 
both experimental and theoretical uncertainties, with the proper error correlation matrix 



B. Neutrino fluxes and interactions 

In order to get significant constraints on the neutrino oscillation hypothesis, the measured 
numbers of /i-like and e-like events produced by neutrinos and antineutrinos in each detector 
must be compared with detailed theoretical predictions. 

The theoretical calculations involve the numerical estimate of integrals of the kind: 



^ We always include both fully and partially contained events in the multi-GeV Kamiokande data 
sample. 



^ We note in passing that the recent reanalysis of the data from the Frejus experiment |32] is 
relevant for the high-energy muon fluxes, but it does not change significantly the previous Frejus 
results [^,^ on low-energy e-like and /U-like events. 

^ The ratio of two normally distributed variables obeys a Cauchy distribution. 
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where 9 is the zenith angle, a and j3 are flavor indices, Ey and Ei are the (anti)neutrino and 
lepton energies, $ is the atmospheric (anti)neutrino flux, P is the oscillation probability, a 
is the (anti)neutrino interaction cross- sect ion, and e is the lepton detection efliciency. Our 
estimate of the /0/3's in the multi-GeV energy range is discussed in Appendix A. 

Concerning the sub-GeV experiments, the ingredients $, o", and e that we use to compute 
the /a/j's have been reported in our previous works [^,^. In particular, we use the Bartol 
group calculations of $ P^ , |5B| for the electron and muon neutrino and antineutrino fluxes 
at each detector location. 

The Bartol fluxes |B3 have been used by all the four sub-GeV experiments in at least 



one simulation, and thus provide us with the advantages of a uniform data analysis and 
of a homogeneous comparison of our calculations with the published detector simulations. 



This comparison has been done in p8| , |29[] . In particular, we refer the reader to Fig. 1 in 
p9| , where our absolute predictions for the lepton energy spectra are superposed to the 
corresponding published Monte Carlo simulations of the Kamiokande, 1MB, Frejus, and 
NUSEX experiments (in absence of oscillations). The agreement of our calculations with 
the published spectra is good. 

Since this work was initiated, new reflned atmospheric neutrino flux calculations have 
appeared PB| , |57| . These new fluxes, as far as we know from the published literature, have 



not been used yet by the experimental collaborations to reprocess their simulations or to 
reanalyze their data, and are not used in this work either. However, in the statistical 
analysis we conservatively associate a ±30% uncertainty (Icr) with the absolute theoretical 
neutrino fluxes. This error accounts conservatively for the spread in the atmospheric neutrino 
calculations published so far |Ty]. The uncertainties of the /i and e neutrino fluxes are 



highly correlated (see |10] and Appendix B for more details). Such correlation is even 



more important than the absolute magnitude of the errors themselves in driving the flts to 
atmospheric v data. In fact, it will be shown that a reduction of the flux error from our 
default value, Ufiux = 30%, to (jfiux = 20% or even 15% does not change signiflcantly the 
results of the oscillation flts. In the absence of oscillations, it has already been shown in 
Ref. |]lO| (Table III) that the statistical signiflcance of the anomaly does not change much 
by reducing the flux error from 30% to 20%. 

The last ingredient of Eq. (|lD to be discussed is the oscillation probability P^p. This is 
the subject of the next Section. 



III. THEORETICAL FRAMEWORK 

The calculation of the atmospheric neutrino oscillation probabilities requires a well- 
deflned theoretical framework. The three-flavor framework used in this paper is completely 
specifled by the neutrino spectrum shown in Fig. 1. 

In Fig. 1 we show the adopted spectrum of neutrino mass eigenstates {yi, z/2, vz)- Two 
states (j/i, V2) are assumed to be almost degenerate in mass. The third state, z/3, is largely 
separated in mass from the almost degenerate doublet, with \m\ — m^2l — being the 
dominant square mass difference driving the atmospheric neutrino oscillations. This situa- 
tion can be realized either in scenario (a) or in scenario (b) of Fig. 1, that is, with z/3 either 
lighter or heavier than vi and z/2. 
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In both scenarios of Fig. 1, the subdominant square mass difference between z/2 and z/i, 
(5m^ = m| — , is assumed to be too small to produce detectable effects in the energy range 
explored by current atmospheric neutrino experiments. This assumption holds, for instance, 
if the parameter 5m? is used to fit solar neutrino data |jl6| {5m^ ~ 5 x 10~^ eV^ at the 



best-fit point for matter-enhanced oscillations). At the scale of the atmospheric neutrino 
experiments, we simply set 5m? = 0. The limits of this approximations are commented 
in Sec. V C. The same approximation has been used in |TB| where we studied accelera- 



tor and reactor neutrino oscillations, and to which we refer the reader for further details 
and references. We collectively consider the accelerator, reactor, and atmospheric neutrino 
experiments as "terrestrial" oscillation experiments. 

At zeroth order in 5m^ /m^ the two scenarios of Fig. 1 are physically different for at- 
mospheric neutrinos propagating in the earth matter (see Appendix C), while they are not 
distinguishable either by means of accelerator and reactor neutrino oscillation searches (in 
vacuum) |T3] or by solar neutrino oscillations (in vacuum or in matter) In this sense. 



atmospheric neutrinos potentially provide a unique information on the neutrino spectrum. 
Notice that the spectra (a) and (b) in Fig. 1 are simply related by: 

(a) ^ (b) +m^ -m^ . (2) 

In the following, we will explicitly distinguish cases (a) and (b) whenever necessary. 

In the framework characterized by the spectrum of Fig. 1 one has two important sim- 
plifications with respect to the most general three-flavor oscillation scenario (see [15] and 
Appendix C): (1) neutrino mixing in terrestrial oscillation experiments can be described by 
just two mixing angles, il) and 0; and (2) effects related to the a possible CP violating phase 
are unobservable. It follows that the parameter space for terrestrial (including atmospheric) 
three-flavor neutrino oscillations can be described in terms of three parameters: (m^, 0). 



The angles il) and are defined as in [38|. In the standard parametrization of the mixing 



matrix |39| the following identifications hold: 

^ = ^23, = ^13 • (3) 

The three-flavor framework with one dominant square mass difference is the simplest 
extension of the two-generation formalism in which all the two-flavor oscillation channels 
are open. The subcases of pure two-flavor oscillations z/g z^r, ^ i/g, and Vy, ^ Ur, 
are recovered in the limits = 0, ^ = 7r/2, and = respectively. A clear graphical 
representation of the general three-flavor case and of its two-flavor limits is provided by 
Fig. 2 in |T^. Here we do not pay attention to the ^ subcase, since it does not 
solve — but rather aggravates — the atmospheric neutrino anomaly. We will briefly comment 
on a theoretically interesting, genuine three-flavor subcase of this framework, the so-called 
threefold maximal mixing scenario P0| , ^ , in Sec. V. In our notation, threefold maximal 
mixing corresponds to (tan^'0, tan^0) = (1, 1/2) at any m^. 

With a neutrino mass spectrum as in Fig. 1, the calculation of the vacuum oscil- 
lation probability for neutrinos arriving from above the horizon is straightforward (see 
Appendix C). However, the vacuum approximation is not adequate for neutrinos which 
travel in the earth matter for a large fraction of their path length. For these neutrinos 
one expects substantial deviations from the vacuum oscillation probability when 
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~ 2\/2GpN ^Ey, where N^. is the typical electron density along the neutrino trajectory. 
This implies significant matter effects in the range iv? ~ 10~^-10~'^ eV^ for sub-GeV ob- 
servables and in the range ~ 10~'^-10~^ eV^ for multi-GeV observables.0 Moreover, 
a genuine three-fiavor effect takes place [2^: the effective mass eigenstates z/J 2 in matter 



are not exactly degenerate, and the phase variation associated to their splitting is relevant 
for path lengths comparable to the earth radius. This additional phase disappears in the 
two-flavor subcases, since one of the "degenerate" neutrinos decouples. 

We take into account the earth matter effects by solving numerically the neutrino propa- 



gation equations with an assigned earth density proflle ^^l- order to save computer time, 
the density proflle is modeled as a 5-step function ||2^ with steps corresponding to the five 
relevant radial shells. This approximation is sufficiently accurate for our purposes. 



We have a final remark on the graphical representations of the results. In |]T3[ we 
have shown that the three-fiavor parameter space for terrestrial neutrino oscillations can 
be usefully charted using the (logarithmic) coordinates (m^, tan^-?/', tan^0). In the fol- 
lowing, the results of our analysis will be displayed in several plane sections of the space 



^ tan^ tan^ 0) at fixed values of either m?, 0, or ip. 



IV. TWO-FLAVOR ANALYSIS 

In this section we show the results of our analysis in the subcases of pure i/^ <-> Ur and 
T-'^i ^ two-fiavor oscillations. The discussion of these cases is interesting in itself, and 
helps in understanding the more complicated situation of three-fiavor oscillations. 

A. Pure ^ v^. oscillations 

In our framework we recall that the subcase of pure z/^ ^ oscillations is reached in 
the limit — > 0, which leaves (m^, tl)) as relevant variables. This subcase is particularly 
simple from a theoretical viewpoint, since is decoupled and thus the earth matter does 
not affect the oscillations. As a consequence (see Appendix C), the scenarios (a) and (b) in 
Fig. 1 become physically equivalent, and the formalism is invariant under the substitution 
'i/' ^ I — -0. One usually takes advantage of this symmetry to restrict the range of the 
mixing angle to [0, vr/4]. Here we show the full range i\} G [0, 7r/2] in order to mark the 
difference with the z/^ ^ case (Sec. IV B), where the symmetry -0 — | — ■?/' is broken by 
matter effects. 

In Fig. 2 we show the results of our analysis as bounds in the (tan^ -0, m^) plane at 90% 
C.L. (solid lines) and 99% C.L. (dotted lines), which corresponds to Ax^ = 4.61 and 9.21 
respectively (A^df = 2). Notice the mirror symmetry of all the curves with respect to the 
axis ?/' = 7r/4. 



^ It should be noted, however, that matter effects do not totally disappear even in the limit 
rr? —>■ 00, see Appendix C. 
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To avoid ambiguities, the allowed regions are marked by stars. We refrain from showing 
the best-fit points in each panel of Fig. 2, since the is often relatively flat around the min- 
imum (except for the combination of several data), and the best-fit point is not particularly 
informative. We give the best-fit coordinates only for the combination of all data. 

In the first four panels of Fig. 2 (left to right, top to bottom) we display the parameter 
regions individually allowed by the four sub-GeV experiments, Frejus, NUSEX, 1MB, and 
Kamiokande. The Frejus and NUSEX experiments are consistent with no oscillations, and 
thus strongly disfavor the situation of maximal mixing [ip ~ vr/4), at least for not too small 
m? where they are not sensitive to oscillations. The 1MB and Kamiokande experiments are 
instead compatible with large mixing in a wide range of m? {m? > 10"^ eV^), ahhough the 
best fit is not reached exactly for maximal mixing. 

It should be noted that the sub-GeV experiments have still some sensitivity to values of 

as small as 10~^ eV^ (see, e.g., |lT7| , p5| , |29[| ) . For instance, the neutrino phase variation 



due to oscillations can be of C(l) for = 1 GeV, = 10~^ eV^, and a path length 
equal to the earth diameter. This fact, combined with our conservative error estimates, 
explains the extension of the 1MB and Kamiokande allowed regions down to values as small 
as ~ 10~^ eV^ in Fig. 2. Our Kamiokande allowed regions are larger than those derived 
by the Kamiokande collaboration analysis . However, one has to consider that we use only 
the published experimental information, while the Kamiokande collaboration uses energy- 
angle lepton distributions that are not pubhshed, and also adopts a rather different approach 
in the statistical analysis of the data [Q. 

The 5th and 6th panels in Fig. 2 show the constraints from the multi-GeV Kamiokande 
data, taken both unbinned and binned respectively. The sensitivity to small is lower 
than in the sub-GeV cases, since the average neutrino energy is higher. In the binned case 
(full information from the angular distribution), is bounded from above at 90% C.L. In 
fact, the measured angular distribution is inconsistent with the fiat oscillation probability 
corresponding to averaged fast oscillations (m^ oo) 

In the 7th and 8th panels of Fig. 2, we combine the sub-GeV and multi-GeV (unbinned 
and binned) Kamiokande data. In the 9th panel all sub-GeV data are combined together. 
Notice that maximal mixing [ip ~ 7r/4) is strongly disfavored for ^ 10~^ eV^, due to 
the inclusion of the Frejus and NUSEX data in the fit. In the 10th panel we combine the 
sub-GeV data with the multi-GeV unbinned data. The multi-GeV unbinned data disfavor 
the lowest values of that would be allowed by the sub-GeV data alone. 

In the 11th panel we combine sub-GeV and binned multi-GeV data. Therefore, this panel 
contains the maximum information from the atmospheric neutrino experiments (8 sub-GeV 
+ 10 multi-GeV observables) . The best-fit is reached at (m^, tari^ ip) ^ (5 x 10^^ eV^, 0.63) 
and at the symmetric point {m?, taia^ ip) ~ (5 x 10~^ eV^, 1/0.63). The best-fit value of 
is somewhat lower than the popular value 10~^ eV^, since we include the Frejus and NUSEX 
data. These data are compatible with = (no oscillation) and thus tend to drag the 
m^-fit to lower values than those favored by 1MB and Kamiokande alone. The value of 
the absolute the minimum is 19.7, which represents a good fit to the 18 atmospheric 

observables (with the freedom to vary only (m^, One should compare this value with 
the corresponding (worse) fit in the no-oscillation hypothesis, Xno osc = 44.6. 

The last panel in Fig. 2 shows the region allowed by the combination of all the established 
accelerator oscillation searches. The region allowed at 90% by all atmospheric neutrino data 
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is also allowed by present accelerator data (reactor data place no bounds in the z/^ ^ Vf. 
oscillation limit). The atmospheric bounds are in conflict with the accelerator bounds only 
at ~ 99% C.L. and high m^. 

In Fig. 2, the magnitude of the theoretical neutrino flux error, ctaux, has been taken equal 
to 30% (default value). This choice is not decisive in the fit, as shown in Fig. 3. 

In Fig. 3 we compare two representative fits (all sub-GeV data, and all sub-GeV + multi- 
GeV data) using both cxflux = 30% (solid contours) and cxflux = 15% (dotted contours). The 
differences between the two cases are very small. In fact, since we include the proper 
correlations between the neutrino fluxes, the v^. and flux errors nearly "cancel" in the 
analysis, with a residual ±5% difference allowed in the relative z/^/z/g flux normalization. 
Of course, a similar cancellation is reached by using the ratio -R^/e = (/i/e)data/(/u/e)theory, 
but with the serious disadvantage of obtaining a highly non-gaussian distribution for i?^/e 
10| . Previous analyses of the atmospheric neutrino anomaly that applied gaussian statistics 
to -R^/e, including our works and may thus have overestimated the statistical 



significance of the anomaly, and underestimated the mass-mixing regions allowed by the 
oscillation hypothesis. 

In this work we have not analyzed the so-called upward-going muon data. These data 
are essentially consistent with no oscillations although with large, non-cancelable flux 
errors — there are no "upward-going electrons" to be used for comparison. Thus we expect 
that, in a combined analysis: (1) the upward-going muon data should have a smaller statis- 
tical weight than the atmospheric data considered here, which should drive the fit; and (2) 
the inclusion of upward muon data should anyway disfavor a too strong suppression of the 
muon rates. This implies that the fit discussed in this section should be generally worsened 
for nearly maximal mixing (■?/' ~ vr/4). The three-flavor analysis of upward-going muon data 
is in progress and will be presented in a separate paper. 



B. Pure <-> Vf. oscillations 

We recall that, in our framework, the subcase of pure ^ v^. oscillations is reached in 
the limit i\} — >■ 7r/2, which leaves (m^, 0) as relevant variables. 

This limit is more complicated than the z/^ *^ limit, since matter effects are not 
decoupled. It follows that the intervals G [0, 7r/4] and G [vr/4, 7r/2] are not physically 
equivalent, and the scenarios (a) and (b) shown in Fig. 1 are also not equivalent. However, in 
the Vp ^ Vf, oscillation case there is an interesting symmetry: the physics in (b) is equivalent 
to the physics in (a), provided that is replaced by its complementary angle (0 ^ | — 0). 
This symmetry (discussed in Appendix C) allows us to consider only one scenario, (a) for 
definiteness. 

In Fig. 4, we show the results of our z/^ ^ z/g oscillation analysis in scenario (a). The 
corresponding results in scenario (b) are obtained by looking at the same figure in a mirror. 
Since the panels in Fig. 4 (z/^ case) are analogous to the panels in Fig. 2 [y^ ^ 

case) we just highlight the differences between the results in Figs. 4 and 2. 

At large m^, the maximal mixing value ^ 7r/4 is in general disfavored, since "too 
many" z/^'s oscillate into z/e's, the muon (electron) rate become too suppressed (enhanced), 
and thus the flavor anomaly is overbalanced by the oscillations. In Fig. 2 this situation 
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was also disfavored, but not as strongly, since in the z/^ ^ Ur case the electron rates are 
unaffected by oscillations. 

At small m^, the contours are affected by the earth matter effects that are also responsible 
for the asymmetry with respect to the axis = 7r/4. Notice that, in general, values of m? 
close to 10^^ eV^ are more disfavored than in Fig. 2. 

The combination of all data is shown in the 11th panel of Fig. 4 (all sub-GeV and multi- 
GeV binned). The allowed region should be compared with the constraints coming from 
accelerator and reactor searches. These constraints essentially exclude the upper part of the 
region preferred by atmospheric v data, but are compatible with the (larger) lower part. 

The best-fit for the combination of all data is obtained for (m^, tan^ 0) = (6.6 x 
10"3 eV^ 0.36). For scenario (b), the best-fit would be obtained at the symmetric point 
^ tan^0) = (6.6 x 10"^ eV^, 1/0.36). The value of at the minimum is Xmin = 15.6, 



{m 

about 4 units lower than in the case of ^ oscillations (Xmin — 19.7). Therefore ^ v^. 
oscillations appear to be preferred to ^ oscillations in our global analysis. 

The preference for the ^ z/e case is essentially driven by the multi-GeV data. In Fig. 5, 
we show the measured and expected /i-like and e-like event rates in the five zenith-angle bins 
used by the Kamiokande collaboration for the multi-GeV data. The /i and e rates have been 
conventionally divided by their theoretical (central) value in absence of oscillations, yUo and 
eo (see also Ref. |jTO| where we introduced this graph). The ellipses represent Icr contours 
(Ax' = 1). 

The first of the horizontal panels in Fig. 5 refers to the no-oscillation case, which is clearly 
a bad fit to the data. The second panel refers to the best-fit to multi-GeV data with pure 

^ oscillations. In this case, only the theoretical muon rates can vary, and the fit is not 
particularly good, although certainly better than in the no-oscillation case. The third panel 
refers to pure ^ v^. oscillations. In this case, both /i and e rates vary with oscillations, 
and one can get higher theoretical electron rates that match better the experimental data 
(notice in particular the good fit in the first bin). The fourth panel refers to the general case 
of three-flavor oscillations, with (m^, t/^, 0) unconstrained. The 3z/ best-fit is only slightly 
better than in the z/^ Vf. case. The differences between the overall fits in the third and 
fourth panels are only appreciable numerically and not by eye. In conclusion, z/^ 
oscillations seem to provide a close-to-optimal fit to multi-GeV data. 

In Fig. 6 we illustrate the importance of including the earth electron density in z/^ ^ z/g 
oscillations. Figure 6 is analogous to Fig. 4 but without matter effects. The allowed regions 
in Fig. 6 and Fig. 4 differ considerably for low m^; when matter effects are included (Fig. 4) 
the lowest values of do not provide a good fit |4^. The reason is that for not too small 
'^fj, ^ mixing (i.e., for tan^ ~ 0.1-10), the mixing angle in matter is rapidly suppressed 
for — > [see Eq. (C4)], and so are the oscillations that should solve the flavor anomaly. 



V. THREE-FLAVOR ANALYSIS IN THE (m^, tan^V', tan^ 0) PARAMETER SPACE 

In this section we show the results of our analysis of atmospheric neutrino data within the 
three-flavor framework discussed in Sec. III. The free parameters of the fit are (m^, ■?/', 0). 
The analysis includes all the sub-GeV data and the binned multi-GeV data. We represent 
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the results in the mixing-mixing plane (tan^V') tan^ 0) at representative values of m^, in 
both scenarios (a) and (b) of Fig. 1. 



A. Scenario (a) 

In Fig. 7 we show the results of the fit to all the atmospheric data in scenario (a). 
The solid (dotted) lines represent sections, at given values of m^, of the three-dimensional 
(m2, tan2 ^, tan^ 0) manifold allowed by the fit at 90% C.L. (99% C.L.) for Nj^f = 3 (Ax^ = 
6.25 and 11.34 respectively). 

The representative values of range from 0.18 eV^ down to 3.2 x 10~^ eV^. For 

^ 10~^ eV^ (first six panels), reactor and accelerator neutrino oscillation experiments 
also place bounds on the mixing angles. These bounds are discussed separately in Sec. V C. 

We recall that in each panel of Fig. 7 the right-hand side corresponds (asimptotically) to 
the limit of pure z/^ z/g oscillations, and the lower side to pure z/^j ^ z/^ oscillations. The 
left-hand side, corresponding to pure z/g ^ z^t oscillations, never represents an acceptable fit 
to the data. The asymptotic regime is already reached at the ends of the tan^ ip and tan^ 
ranges adopted in Fig. 7. 

The best three-flavor fit is reached at (m^, tan^V^, tan^^) = (4.6 x lO'^ eV^ 7.07, 0.28). 
The corresponding value of the is Xmin — 14.8, which represents a good fit to the 18 
atmospheric observables, given the freedom of varying the three parameters (m^, tp, 0). 

At 90% C.L. there are both an upper and a lower bound on m^: 0.6 x 10~^ ^ ^ 
1.5 X 10~^ eV^. The upper bound, provided by the inclusion of multi-GeV data, however, 
disappears at ~ 95% C.L. (see below). The inclusion of laboratory oscillation data would 
make the upper bound tighter (m^ ^ 6 x 10~^ eV^ at 90% C.L., see Sec. V C). 

For relatively large (m^ ^ 2 x 10^^ eV^) the situations of maximal z/^ ^ z/g mixing 
[(tan^'0, tan^0) = (oo, 1)], of maximal z/^ ^ z/^ mixing [{tan^ip, tan^0) = (1, 0)], and of 
threefold maximal mixing [(tan^'0, tan^ 0) = (1, 1/2)], are not allowed. All these twofold 
and threefold maximal mixing situations are allowed, however, in the range 1.5 x 10^^ ^ 



m 



2 < 7 X 10"^ eV^ (at least). In this range, the vacuum oscillation probabilities for threefold 



r\-/ 



maximal mixing P0| , ^ get significant corrections when matter effects are included (see 
Appendix C). 

In many panels of Fig. 7, the allowed region interpolates smoothly between the two- 
fiavor oscillation limits Vj. and z/^ ^ z/g [2C-23,15]. However, for ^ 2 x 10^^ eV^, 



pure z/^ Vr oscillations are disfavored since the global fit improves towards the ^ v^. 
oscillation limit. 

The limits on vn? for unconstrained and are particularly interesting as guidelines for 
future long-baseline neutrino oscillation searches. In Fig. 8 we thus show the value of Ax^ 
as a function of only (-0 and are projected away). The solid line refers to the default 
flux error (ctaux = 30%). The dashed line, which refers to ctaux = 20%, is not significantly 
different from the solid line. 

From Fig. 8 one can trace the upper and lower bounds on vn^ placed by all the atmospheric 
neutrino data at any given C.L. However, for vn? oo the A^^ tends to the asymptotic 
limit ~ 7 (not shown). It follows that, in the adopted three-flavor framework, atmospheric 
neutrinos place no upper bound on vn? at 95% C.L. (A^df = 3). Atmospheric neutrino 
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data would also not place any upper bound on if the zenith-angle dependence of the 
multi-GeV data were discarded, i.e. if unhinned multi-GeV data were used in the fit. 



B. Scenario (b) 

In Fig. 9 we show the results of the fit to all the atmospheric data in scenario (b) of 
Fig. 1. Figure 9 is analogous to Fig. 7, but all the calculations have been done with —rn? 
instead of +m^. The solid (dotted) lines represent sections, at given values of — m^, of the 
three-dimensional (m^, tan^ ip, tan^ 0) manifold allowed by the data at 90% CL. (99% CL.) 
for Ndy = 3. 

In scenario (b), the best-fit point is (m^, tan^ ^z^, tan^ 0) = (6.8 x 10"^ eV^, 11.2, 2.82). 
The corresponding value of xLin is 15.1, which is almost as good as in scenario (a). 

As expected from symmetry arguments (see Appendix C), Fig. 9 and Fig. 7 coincide 
in the limit of pure z/^ ^ z/,- oscillations (lower side of each panel). In the limit of pure 
t^fi ^ oscillations (right side of each panel) these figures coincide]^ modulo the replacement 
— >■ I — 0. In the intermediate, genuine three-fiavor mixing cases, the bounds shown in 
Figs. 9 and 7 are slightly different at any m?. 

The differences between the three-flavor flts in Figs. 7 and 9 are not unexpected, since 
they correspond to two physically different scenarios. Unfortunately, the differences are quite 
small, implying that the available information on atmospheric neutrinos is not sufficiently 
accurate to discriminate the two cases (a) and (b). A significant discrimination would have 
important implications. For instance a hypothetical, pronounced preference of atmospheric 
data for scenario (a) would support the theoretical prejudice that the spectrum of neutrino 
masses is similar to the spectrum of charged fermions (two light states and a third, much 
heavier state). It will be interesting to see if the atmospheric neutrino data that are being 
collected with high statistics by the running SuperKamiokande experiment will show a pref- 
erence for one of the two scenarios (if they confirm the fiavor anomaly). We recall that this 
information cannot be provided either by accelerator ,[] reactor, or solar neutrino oscillation 
searches, i.e., these experiments a priori do not distinguish the scenarios (a) and (b) at 
zeroth order in 5m^ /m?. 

Finally, we complete our survey of the fit in scenario (b) by showing in Fig. 10 the 
dependence of Ax^ on —m?. Fig. 10 is the analogous to Fig. 8 in scenario (b). 



^ The coincidence of the CL. contours in these two-flavor limits is not perfectly realized because 
the best-fit point and the value of the at the minimum are not exactly equal (and are not 
expected to be equal) in scenarios (a) and (b). 

^ However, futuristic accelerator oscillation searches with extremely long baselines (greater than 
10^ km) could in principle probe the difference between scenarios (a) and (b) through earth matter 
effects. 



11 



C. Comparison with other oscillation searches 



The three-flavor bounds shown in Figs. 7 and 9 and discussed in Sections V A and V B 
were obtained by fitting only the atmospheric neutrino data. In this section, we show their 
interplay with the independent constraints obtained by accelerator and reactor neutrino 
oscillation searches [jl5[ and by solar neutrino experiments ||16|| . 



The analyses [|1^ and ||16[ were performed under the same assumption on the neutrino 
spectrum shown in Fig. 1, namely that the two independent neutrino mass squared differ- 
ences, 5m? = \m\ — m\\ and im? = \m1 —mW, are largely separated: 5m? -C m^. Accelerator 
and reactor neutrinos were assumed to probe, as the atmospheric neutrinos, the dominant 
square mass difference m^, as the slow oscillations driven by 5m^ were effectively frozen. 
Conversely, solar neutrinos were assumed to probe the subdominant square mass difference 
5m^, as the fast oscillations driven by m^ were effectively averaged out. The parameter 
spaces probed by terrestrial (accelerator, reactor, and atmospheric) neutrinos and by solar 
neutrinos have been discussed thoroughly in and . 

In Fig. 11, we show the bounds coming from the established accelerator and reactor 
oscillation searches (the recent data from the Liquid Scintillator Neutrino Detector 
(LSND) experiment are not included). These negative searches exclude horizontal bands 
in the {ian^ip, tan^0) plane for ^ 10~^ eV^ (90% CL. limits are shown). Superposed 
are the 90% CL. regions from the atmospheric neutrino analysis (only in scenario (a) for 
definiteness) . The ^ z/g oscillation limit (right side of the panels) is generally disfavored 
in the range probed by laboratory oscillation experiments, mainly because it is not consistent 
with the unsuccessful Ve disappearance searches at reactors. For m^ ^ 6 x 10~^ eV^ the 
atmospheric data are not compatible with existing laboratory limits at any or 0. For 

^ 10~^ eV^, however, there are no laboratory constraints on neutrino mixing, and the 
fits to atmospheric data (six lowest panels of Figs. 7 and 9) are unaffected. 

In conclusion, the limits placed by the combination of accelerator, reactor, and atmo- 
spheric neutrino data on m^ in our three-flavor framework are 6 X 10"^ eV^ < m2 < 6 X 10"^ 
eV^ (90% C.L., A^DF = 3). Future long-baseline experiments will be able to explore a large 
fraction of this m^ range. 

Concerning solar neutrinos, we have emphasized in [|16| that they probe the same mixing 
angle (f) probed by terrestrial (atmospheric, accelerator and reactor) neutrino experiments. 
If one accepts the explanation of solar neutrino deficit provided by matter-enhanced oscil- 
lations, then the data constrain cf) in the range tan^ ^ 1.4 at 90% CL. (A'df = 3), with 
a preference for the value tan^ = |]16[ . These bounds coming from solar v data exclude 
significant parts of the large-0 regions allowed by atmospheric neutrino data in Figs. 4, 7, 
9, and 11. This should be taken into account when building models of neutrino masses 
and mixings which try to accommodate both the solar neutrino deficit and the atmospheric 
neutrino anomaly. 

In our framework, once 5m^ is fixed by solar neutrinos, it is not possible to explain both 
the atmospheric neutrino anomaly and the recent LSND evidence PU| for oscillations with 



the remaining mass parameter m^. We have shown previously that atmospheric neutrino 
data alone place an upper limit to m^ of about 1.5 x 10~^ eV^, which is strengthened to 
m? ^ 6 X 10~^ eV^ when accelerator and reactor data are included. The range ^ 6 x 10~^ 
eV^ is too low for significant neutrino oscillation effects at LSND. It follows that, in our 
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framework, one can either fit the atmospheric neutrino anomaly or the LSND event excess, 
but not both. We intend to examine the second option (fit to LSND data) in a separate 



pubhcation; some interesting results were already obtained with older LSND data in |T5 
(see there Figs. 11 and 12 and the related discussion). However, it should be noted that the 
atmospheric data fit at large is essentially driven by the angular distribution of multi- 
GeV events observed in Kamiokande. If this information were discarded, i.e., if one used only 
unbinned multi-GeV data, then atmospheric neutrino data alone would not place an upper 
bound on m^, and one could find |Q a very small region of the parameter space which 
is marginally allowed by both LSND and sub-GeV atmospheric data, as well by present 
accelerator and reactor constraints. This solution is admittedly fragile [pO|]. 



A final remark is in order. The basic assumption underlying this work and Refs. 1151116 



is that 5m? -C (Fig. 1). Then all the calculations are done at zeroth order in bvn? jw?^ 
i.e., one takes bm? = and finite for terrestrial neutrino oscillations, and Sm"^ finite 
and = oo for solar neutrino oscillations. If is close to the best-fit to solar neutrino 
data ((5m^ ~ 5 x 10^® eV^) and if is close to the best-fit to atmospheric neutrino data 
(m^ ~ 0.6 X 10~^ eV^) then 5m^/m^ ~ 10"'^ and the zeroth order approximation is certainly 



adequate. In Ref. p9| we have numerically shown that for 5m /m as high as 1/10 the 



leading first-order corrections to the zeroth approximation do not alter substantially the 
results of both the solar and the atmospheric neutrino data fit. However, if one takes the 
highest values of allowed by solar neutrinos {5m^ ~ 1.5 x 10~^ eV^) [1^ and the lowest 



values of m^ allowed by atmospheric neutrinos (m^ ~ 6 x 10^"^ eV^) at 90% C.L., then the 
ratio dm^ /m^ is about 1/4, so that the two squared mass differences are not well-separated, 
and our approximations become very rough. Such a contrived situation seems improbable, 
but if it were realized in nature, then one should necessarily resort to the most general 
three-flavor formalism to analyze it. 



VI. SUMMARY AND CONCLUSIONS 

In this work we have analyzed the available experimental data on the anomalous flavor 
composition of the atmospheric neutrino flux with three-flavor neutrino oscillations. Data 
on upward-going muons are not included and will be examined in a separate work. 

The adopted theoretical framework is characterized by one dominant neutrino square 
mass difference, m^. The neutrino mass spectrum can assume either form (a) or (b) of 
Fig. 1. Scenarios (a) and (b) are physically different when neutrino oscillations in the earth 
matter background are considered. In both cases the variables relevant to atmospheric 
neutrino oscillations are m^ and two mixing angles, and 0. 

We have performed a global analysis of all data, and found the regions of the (m^, ip^ 0) 
parameter space in which three-flavor neutrino oscillations are consistent with the available 
data. In particular, we have included in the analysis the neutrino-induced e-like and /i- 
like event rates measured in four sub-GeV experiments, Frejus, NUSEX, Kamiokande, and 
1MB, as well as the lepton rates measured in the five zenith-angle sectors of the multi-GeV 
Kamiokande experiment, for a total of 18 observables in the fit. We have made accurate 
calculations of the expected muon and electron rates in the various detectors, taking into 
account the differential energy-angle distribution of the (anti)neutrino fluxes, the differential 
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(anti)neutrino interaction cross-sections, and the detector efficiencies. We have paid partic- 
ular attention to the statistical analysis, which includes the proper correlations among the 
experimental and theoretical errors. The oscillation probabilities have been calculated in the 
three-flavor framework defined by Fig. 1, including the earth matter effect in the evolution of 
the neutrino flavor states. The main results of our analysis are shown in Fig. 7 for scenario 
(a) and in Fig. 9 for scenario (b). Three- flavor neutrino oscillations provide a good fit to 
the 18 data (with three free parameter), as the minimum is ~ 15 in both cases. 

We have also analyzed in detail the subcases of two-flavor i/^ ^ oscillations (Fig. 2) 
and <-> z/e oscillations (Fig. 4). The analysis of these subcases provided us with valuable 
information on the relative weight and influence of the different pieces of data in the global 
fit, as well as on the importance of the earth matter effect (as derived by comparing Fig. 4 and 
Fig. 6). The Kamiokande multi-GeV data are fitted better in the ^ Ve case than in the 
i/^ z/t- case, as shown in Fig. 5, although more data are needed to confirm this indication. 
In particular, the upcoming data from the running SuperKamiokande experiment will 
certainly help in clarifying the atmospheric neutrino anomaly and its implications in terms 
of neutrino properties. 

We have compared the region preferred by atmospheric neutrino data with the bounds 
coming from negative oscillation searches at accelerator and reactors |TB| in Fig. 11. These 
bounds exclude a large part of the region allowed by atmospheric data (especially in the 
limit of pure z/^ z/g oscillations) for ^ 10^^ eV^. For ^ 10^^ eV^ there are no 
significant bounds from reactor data and the atmospheric data fit is unaffected. In particular, 
for m? % 10~^ eV^ the threefold maximal mixing scenario QjHI] is allowed by the data. It 
must be added, however, that threefold maximal mixing is not supported by the independent 



analysis of all solar neutrino data |16 



In Sec. V C we have also discussed the interplay between atmospheric and solar neutrino 
results. Solar neutrino data place an upper bound on cf) |jl6| that further constrains the 
atmospheric results at any m? . The value of needed to fit atmospheric neutrino data 
is not compatible, within this framework, with the possible recent indication for neutrino 



oscillations coming from the LSND experiment [49|. A marginal compatibility between the 



LSND data and the atmospheric anomaly might be reached if the information coming 
from the zenith-angle distribution of multi-GeV event is discarded. 

This work is part of a wider research programme in which we intend to analyze, in 
the same three-flavor framework, the world data related to neutrino oscillations. We have 
analyzed so far the results coming from 14 experiments: 3 accelerator experiments (CERN 
CDHSW, Fermilab E531, and BNL E776) [1^, 3 reactor experiments (Bugey, Gosgen, and 
Krasnoyarsk reactors) [^, 4 solar neutrino experiments (Homestake, GALLEX, SAGE, 
and Kamiokande) |jl6[, and 4 atmospheric neutrino experiments (Frejus, NUSEX, 1MB, and 
Kamiokande, this work). (See also pSj^ .) We have discussed in some implications of 
older LSND results |^T|, and in ||I4[ the tests of three- flavor mixing in future long-baseline 



neutrino oscillation experiments. We hope that the three-flavor framework adopted in these 
works can become a popular way of analyzing or even presenting the experimental results or 
expectations, instead of the usual two-generation approach which is unable to accommodate 
more than one oscillation channel at a time. 
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APPENDIX A: TREATMENT OF KAMIOKANDE MULTI-GeV DATA 

In this appendix we describe in detail our treatment of the Kamiokande multi-GeV data. 

The analysis of the Kamiokande multi-GeV data |Q depends crucially upon the distri- 
bution of lepton events {Np) of given flavor /5 as a function of the zenith angle 9. In the 
presence of neutrino oscillations, /5-fiavor lepton events may be initiated by neutrinos of 
original flavor a, and the angular distribution can be expressed as: 

dN0 ^ (P^^ r^^ dap 



d9 ^ Je}-^^^ ^ dE,d9 "^7£;r" dE, 



1 



where: 



dNp{9) 



lepton angular distribution 



d9 

By = neutrino energy 
El = lepton energy , 



distribution of unoscillated Ua , (A2) 



dE^d9 

Pap{Ey, 9) = flavor oscillation probability 



dap{Ei) 



differential i^/? cross section 



dEi 

e()[Ee) = lepton detection efficiency . 

For the sake of simplicity (and of computing time) in Eq. ( |Al| ) we have assumed that 
the lepton direction 9 is the same as the incident neutrino direction 9,^, 9 ~ 9,y. Actually, 



in the Kamiokande multi-GeV data sample, the typical difference is y {{9 — 9i,Y) 15°-20° 
Q. We simulate the effect of the 9-9i, difference by smearing the distribution dNp/d9 in 
Eq. ([Al|) with a Gaussian distribution having a one-sigma width of ~ 17°. 



The evaluation of the inner integral in Eq. ( [Al] ) requires detailed experimental infor- 
mation that is not available. In particular, the lepton detection efficiency function ep{Eii), 
which includes the intrinsic detector acceptance and the analysis cuts, is not published for 
multi-GeV data A worse problem is due to the impossibility of defining, event by event, 
the lepton energy E^ for tracks that are not fully contained. For these (partially contained) 
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higher-energy events, the intrinsic "total" energy can be associated with the released "vis- 
ible" lepton energy on a statistical basis only, by means of a detailed simulation of the 
Kamiokande detector (which is beyond our possibilities and interests). However, the igno- 
rance of such experimental ingredients can be overcome by re-writing Eq. (|A1|) in terms of 
the simulated energy spectrum of the parent neutrinos (which embeds all these effects) as 
published in 

The method is the following. The {E,y, ^)-distribution of unoscillated neutrinos can be 
factorized as: 

rf2$„(E„ e) d^iE,) d^';,{E,, e\ ^^^^ 



dEy de dEv d9 



dO "^"^"^f = 1 (at any E,) . (A4) 



with the normalization 



dd 

Applying the factorization of Eq. ( [A3| ) in Eq. ( [Al[ ) one has that: 

'dEu -JT Pap -TET dE, — Bp 



where 



dO J d9 dEj ' dE, 

dE.^Pa,Qap^^, (A5) 



is known from atmospheric flux calculations, and 



dn,(E.)_dt', f^^^^^^ (A7) 



dE^ dE^ J dE, 

represents the energy distribution of the parent neutrinos that induce /5-lepton events in the 
Kamiokande detector, integrated over the lepton spectrum. 

As reference energy-angle neutrino flux distributions, we use the calculations of the 
Bartol group [^] smoothly connected to the Volkova calculations |Q at higher energies. 
This reference choice corresponds to the option "Flux B" in |Q. 

The function dnj3{E^)/dE,y, that embeds all those experimental aspects of the lepton 
detection efficiency and energy reconstruction that we ignore, is published in Fig. 2(b) of 
Ref. [^. Since both d^'^/dO and Qap are known, and Pq,/3 is calculable in a given oscillation 
scenario, one finally has all the ingredients to calculate the angular distribution of lepton 
events from Eq. (|A5|) f] 



It should be added, however, that in this way our analysis of the multi-GeV data depends implic- 
itly upon the neutrino cross sections as implemented in the Kamiokande Monte Carlo simulation. 
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In particular, we have computed the angular distribution of e-like and /x-like multi-GeV 
events in absence of oscillations {Paf3 = Sa/s)- The results are shown in Fig. 12 as dashed 
lines. The solid lines represent the published Kamiokande simulation Q]. The agreement is 
very good, as the differences are smaller than the statistical uncertainties. 

The author of Ref. |^ used independently a somewhat similar approach to the analy- 
sis of multi-GeV data but did not obtain, however, a good agreement with the published 
Kamiokande simulation of e-like events. 

A final remark is in order. The functions dnp{Ey) / dE^ {(3 = e, /i) reported in |^ include 
the contributions of both neutrinos and antineutrinos. However, it is important to separate 
the u and u contributions, since P(z^a 1^13) 7^ Pi^^a ^p) when matter oscillations are 
considered. We make the reasonable assumption that Sp ~ ej^. Then, since the ratios of u 
and v fluxes and cross sections are known at any energy, one can separate the distribution 
dnp/dEy of parent neutrinos from the distribution dn^/dE^, of parent antineutrinos. 



APPENDIX B: CORRELATION OF ERRORS 



In |T0| we examined the various sources of uncertainties affecting the measured and 



expected e-like and /x-like event rates Re and in the atmospheric neutrino experiments, 
together with their correlations. We were thus able, for any single experiment, to construct 
the covariance matrix of the residuals {R^^^°'^ — R'^^^) and to perform a correct (Gaussian) 
statistical analysis of the atmospheric u anomaly. As far as a single experiment is concerned, 
here we use the same approach. 

However, when the information of two or more experiments is combined — as in the 
present work — one has also to take into account that the theoretical errors of the neutrino 
fluxes are highly correlated from experiment to experiment. For instance, a hypothetical 
systematic shift of +20% in the calculated (unoscillated) flux of sub-GeV u^s propagates 
coherently to the expected rates of e-like events in all the sub-GeV experiments at the same 
time. Moreover, one also expects the /x-like event rates to increase by ~ 20% because of the 
tight correlation of calculated and fluxes, with an allowance for a residual uncertainty 
(of about 5%) in the /i/e ratio. 

Therefore, in constructing the covariance matrix for the observables analyzed in this 
work, we include the additional off-diagonal elements corresponding to the correlations of 
the neutrino flux uncertainties between any two experiments, and between any two bins of 
the Kamiokande multi-GeV data sample.^ 

More precisely, let us call {A, B) a generic couple of experiments (or couple of multi-GeV 
data bins). Then the correlations between the Ve and theoretical flux errors in A and B 
are given by: 

Pee{A, B) = l , 



° In principle there could also be correlations among the experimental systematic uncertainties 
affecting any two bins of the Kamiokande multi-GeV data. However, for lack of published infor- 
mation we ignore such additional correlations. 
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p^,,{A, B) = l , 



(Bl) 



p,e{A, B) = 1 




where cTflux is the assumed fractional uncertainty in the overall flux normalization (e.g., 
±30%), and cTfatio is the assumed residual uncertainty in the /i/e ratio (typically ±5%, 
which we choose as default value). For instance, for (ctaux, cTratio) = (30%, 5%) the /i-e flux 
error correlation is p^e = 0.986 |jTD[- 

We have decided, however, to ignore the correlations when A labels a sub-GeV observable 
and B labels a multi-GeV observable (or vice versa). In fact, the flux of low-energy and 
high-energy neutrinos are not necessarily correlated. A systematic shift of, e.g., +20% in the 
low-energy neutrino flux normalization does not necessarily imply the same shift at higher 
energy. This would happen, for instance, if the slope of the theoretical neutrino energy 
distribution were systematically biased. At present, we do not know how to relate the 
uncertainties affecting the low-energy and the high-energy fluxes of atmospheric neutrinos, 
and thus ignore their possible correlations in the statistics. 

The inclusion of the (known) correlation effects in any single experiment, as well as in 
the combination of all the experimental data, is a distinguishing feature of our analysis. 

APPENDIX C: SYMMETRIES OF THE OSCILLATION PROBABILITY 

In this appendix we discuss several symmetry properties of the neutrino and antineutrino 
oscillation probabilities under given transformations of the neutrino masses and mixing, in 
the two scenarios (a) and (b) of Fig. 1. These properties are useful to understand the 
results of the analysis of the atmospheric neutrino data. In particular, we show that the 
scenarios (a) and (b) are not equivalent when the earth matter effects are included in the 
(anti) neutrino propagation. 

We recall that we always assume m? positive (m^ = |m^|), and that the two scenarios 
(a) and (b) are distinguished by the overall sign of m?: (a) (b) <^==^ +m? —m?. It is 
useful to set conventionally the zero of the neutrino squared mass scale halfway between the 
doublet (z/i, 1/2) and the "lone" state 1/3 shown in Fig. 1. With this position, the neutrino 
squared mass spectrum takes the form: 




scenario a 



scenario (b) 



(CI) 



Let us consider the following 8 transformations Tj: 



Ti : m? — i> —m? 



at any ijj, ; 
at '?/' = ; 
aX ip = tt/2 ] 



T2 : (m^, 0) (m^, 7r/2 - 0) 

T3 : (m^, 0) ^ (m2, ti/2 - 0) 

T4 : (m^, ^p) (m^, ti/2 - tp) 

Ts : (m^, 0) ^ (-m2, 7r/2 - 0) 

Te : (m^, 0) ^ (-m^, n/2 - 0) 

Ty: (m^, ?/>) ^ (-m^, 7r/2-^/') 



at = ; 
at ^/^ = ; 



at = ; 
at any -0, 



at = 7r/2 ; 



(C2) 
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The transformation Ti changes the overall sign of and thus maps scenario (a) into 
(b) or vice versa. The transformations T2^5, Ta g, and T4 7 are relevant respectively for the 
subcases of pure two-flavor v^, ^ z/^- oscillations (■?/' = 0), i/g ^ oscillations {ip = ^/2), 
and Ufj^ <-i> Ut- oscillations (0 = 0). The transformation Tg interchanges neutrinos with 
antineutrinos and, at the same time, changes the sign of m^. Notice that the Tj's in Eq. ( |C2D 
are not all independent: T5 = Ti ■ T2, Tg = Ti ■ T3, and T7 = Ti ■ T4. 

We prove the following statements: (1) in vacuum, the oscillation probabilities are in- 
variant under Ti, T2, T3, T4, T5, Tg, T7, and Tg; (2) in matter, the oscillation probabilities 
are invariant only under T4, T5, Tg, T7, and Tg; (3) in matter, the additional symmetries Ti, 
T2, and T3 are restored in the limit ^ 00. 

The statement (1) is evident from inspection of the vacuum oscillation probabilities 
(equal for neutrinos and antineutrinos): 



(C3) 



pvac 
ee 






pvac 
MM 


= 1-444(1- 


4'^V') ^ ' 


pvac 

TT 


= 1-444(1- 


44) ' 


pvac 
efj, 


- 4s2^2 2 




pvac 

er 


- AqW^ 9 




pvac 
MT 







where S = sm'^{m'^x/4:E^). Note that the angles u and 6 never appear in Eq. (|C3|) . Of 
course, Pa/3 = Ppa- 

The probabilities in Eq. ( |U3| ) are invariant under Ti, implying that the cases (a) and (b) 
are indistinguishable in vacuum. The symmetries T2 and T5 imply that the parameters of 
pure Ve ^ Vt oscillations in vacuum can be restricted to the case +m? and (j) G [0, vr/4], as 
the cases —rn? G [vr/4, 7r/2] become equivalent. Analogously, this is true for pure z/g 
or ^ Vr oscillations in vacuum. 

When matter effects are included, the situation becomes more complicated [^] and 
several symmetries are broken. For the sake of simplicity, we discuss the symmetry properties 
of the oscillation probabilities in the case of constant electron density, = const. Our 
conclusions are also valid for a generic N^, = Nf,{x), but the proof is considerably more 
involved and less transparent, since the neutrino propagation equations are not analitically 
integrable for a generic density. Instead, for constant N^, the oscillation probabilities can be 
expressed in compact form. For neutrinos they are given by (we omit the derivation): 



(C4) 



p 

ee 


— 1 — 4s|c| 5*31 , 
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= 1 - 4s|44 '^si 
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= 1 - 4s|44 "^si 
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5'32 
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e/x 


= 4s|c|s^ 5*31 , 
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A 2 2 2 2 Q 
— 4S^C(j,S^C^ O31 


+ 4s|s^4 


5'21 




>S'32 



where $ is the effective mixing angle (j) in matter [ip remains unchanged): 
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sin 2$ = ""''^ . (C5) 

(cos 2(f) T Ajm^Y + (sin 20)2 



In Eq. (|C4|) the oscillating factors Sij are defined as 



S,, = sin' ( -^j^x ) , (C6) 



where are the neutrino square mass eigenvalues in matter, 

^ ^ 2 S2$ 2 ' 
M', = , (C7) 

^ 2 S2$ 2 ' 

and A = 2\/2GFNf,E^ is the matter- induced square mass term. We recall that the neutrino 
square mass eigenvalues in vacuum are given in Eq. (PT]). 

In Eqs. (P5|)-(P7|) the upper sign refer to scenario (a) and the lower sign to scenario 
(b). Notice that, as in the vacuum case, the angles uj and 6 never appear in the oscillation 
probabilities in Eq. ( P4|) . 

From Eqs. (p4|)-(|C7|) it follows that the earth matter effects do not vanish for threefold 
maximal mixing, corresponding to (tan^?/^, tan^ 0) = (1, 1/2). In the interesting range 
10^'^ eV ^ ^ 10^^ eV , the matter corrections to the "vacuum" lepton rates can be as 
large as 10% in the sub-GeV case and as large as 30% in the multi-GeV case. The fits to 
the threefold maximal mixing scenario in [^,0 were performed neglecting matter effects. 



The oscillation probabilities in Eq. ( |U4|) are not invariant under the transformation 
Ti. Therefore, the scenarios (a) and (b) are in general physically different for atmospheric 
neutrinos traversing the earth matter. The difference can be traced to the matter-induced 
neutrino mass term A [Eq. (|C7|) ], which is positive both in (a) and (b), while the overall 
sign of the vacuum mass changes in the two scenarios. 

However, the probabilities in Eq. (|C4|) are invariant under the transformations T4, T5, 
Tq, and T7. Notice in particular that: 





= 




^ Z/r) - 


P 






= 7r/2 






P 


= 4s|,c|,5' , 




= 




^ Ur) - 


P 





(CJ 



pvac 



where 



The symmetry T3 does not apply to the probabilities in Eq. ( |(J4[ ) . Therefore, the de- 
scription of pure two-flavor oscillation Vf. ^ in matter cannot be exhausted by taking 



20 



and G [0, vr/4]. Either one takes and extends the range of to [0, 7r/2] (as 

in this work), or one keeps G [0, 7r/4], but considers both and — (as in [^). 

Analogously, this is true for i/g ^ v^. oscillations in matter, which are not invariant under 

Notice that the symmetry T4, corresponding io if) ^ — dX given for pure ^ Vr 
oscillations (0 = 0), holds both in vacuum and in matter. In fact, matter effects are irrelevant 
for pure z/^ oscillations. 

Let us now consider the antineutrino oscillation probabilities. Equation ( |C3|) holds 
both for neutrinos and antineutrino oscillations (in vacuum). Equation ( |C^ ) refers only to 
neutrinos. For antineutrinos, the matter-induced term A changes sign. However, if one 
also changes the sign of m^, the antineutrino propagation becomes equivalent to neutrino 
propagation. In other words, the oscillation probabilities of neutrinos in scenario (a) are 
equal to the oscillation probabilities of antineutrinos in scenario (b), and vice versa, as 
expressed by the symmetry Tg. Notice that the higher symmetry Ti ■ Tg, which corresponds 
to z/ ^ P at any given (m^, tp, 0), holds in vacuum only. 

It is amusing to notice that, in the purely hypothetical situation of equal u and u fluxes 
at any energy E^, and of equal u and u absorption cross sections, the symmetry Tg would 
imply the same physics in atmospheric neutrino experiments in both scenarios (a) and (b). 

Finally, we consider the case of —>■ 00 or, more precisely, S> A. In this limit, the 
oscillation probabilities in Eq. ( p^ ) become: 

p pvac 

p pvac _ r p 

Prr = - , (CIO) 

p pvac 

p pvac 

p,, = p;^; + 5P , 

where 5P = Aslsf/^c^ sin^ {Aclx / AE^) . 

Notice that in the limit ^ A not all the probabilities in Eq. ( |C10| ) tend to their 
vacuum value. However, in this limit all the symmetries (Ti, T2, . . . , Tg) of the vacuum 
oscillation case apply. In the subcases of pure two-flavor oscillations [ip = ot = it/ 2 or 
= 0) one has 6P = and Eq. ( |C10|) simply reads Pafs = P^^p (averaged vacuum oscillations 
regime) . 
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FIGURES 



FIG. 1. The neutrino mass spectrum adopted in this work. One neutrino mass eigenstate 
(1/3) is assumed to be largely separated from the quasi-degencratc doublet [ui, U2) by a square 
mass difference jml — m\ 2I — rn? ■ The two possible scenarios (a) and (b) arc physically different 
when the earth matter effects are included in the atmospheric v propagation. The two square mass 
spectra in (a) and (b) are related by: (a) (b) <^=> m? — ^ —m?. 

FIG. 2. Analysis of separate and combined atmospheric neutrino data assuming pure ^ Vj- 
oscillations ((^ = in our framework) in the mass-mixing plane (tan^ -0, m^). Contours at 90% and 

99% C.L. for -/Vdf = 2 are shown. The allowed regions are marked by stars. The last panel shows 
the constraints coming from the combination of established accelerator data. Notice the symmetry 
of all the allowed regions with respect to the axis = n/A. 

FIG. 3. Effects of the reduction of the neutrino flux uncertainty from the default value of 
±30% (solid lines) to ±15% (dotted lines). The variations in the regions allowed at 90% C.L. by 
sub-GeV data (upper panel) or sub-GeV+multi-GeV data (lower panel) are very small. 

FIG. 4. Separate and combined analysis of atmospheric neutrino data assuming pure I'f^ 
oscillations = 7r/2 in our framework), in the mass-mixing plane (tan^ (/), m?). Contours at 90% 
(solid) and 99% C.L. (dotted) for A^df = 2 are shown. The allowed regions are marked by stars. 
The last panel shows the constraints coming from the combination of established accelerator and 
reactor data. The contours of the atmospheric v allowed regions are not symmetric with respect 
to the axis <^ = 7r/4, due to matter oscillation effects. 

FIG. 5. Bin-by-bin analysis of multi-GeV Kamiokandc data in the plane of the ^ and e 
lepton rates, normalized to their theoretical values without oscillations, /iQ and eg. Solid ellipses: 
theoretical predictions at la level (Ax^ = 1). Dotted ellipses: experimental data at la level. 
Notice how the theoretical ellipses change from the upper panel (no oscillation) to the three lower 
panels (best-fit cases for two-flavor and three-flavor oscillations). The fit is better in the ^ 
case than in the ^ v-r case. The overall fit improves slightly in the 3z/ oscillation case. 

FIG. 6. As in Fig. 4, but excluding the earth matter effect (pure vacuum oscillations). All 
the contours are now symmetric with respect to the axis (j) = 7r/4. The regions allowed by the 
atmospheric neutrino data in the lower half of each panel are substantially different from those 
reported in Fig. 4. 
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FIG. 7. Three-flavor analysis of all the atmospheric neutrino data (sub-GeV and binned 
multi-GeV combined) in the plane (tan^ ^, tan^ (p), for 12 different values of ranging from 
1.8 X 10"^ to 3.2 X 10"^ eV^. Scenario (a) of Fig. 1 is assumed. The solid (dotted) curves represent 
sections of the region allowed at 90% (99%) C.L. for N-£,f = 3 at given m^. The right side of each 
panel corresponds asymptotically to pure V/^ oscillations; the lower side to pure ^ Vr 

oscillations. Three-flavor oscillations interpolate smoothly between these two limits. 

FIG. 8. Value of Ax^ for all atmospheric neutrino data (sub-GeV and multi-GeV combined) 
as a function of only. This figure embeds the information of Fig. 7 projected onto the 
variable. At 68% C.L. (A^df = 3) the value of is constrained between ~ 10~^ and ~ 10~^ eV^. 
For very large m^, the value of Ax^ tends to ~ 7 (not shown) and there are no upper limits on 

at 95% C.L. The reduction of the v flux error from 30% to 20% does not produce significant 
variations, as indicated by the thin, dotted curve. 

FIG. 9. As in Fig. 7, but in the scenario (b) of Fig. 1. 

FIG. 10. As in Fig. 8, but in the scenario (b) of Fig. 1. 

FIG. 11. Comparison between the regions allowed at 90% C.L. {Ndf = 3) by the atmospheric 
neutrino data in scenario (a) (solid contours), and the corresponding regions excluded by the 
established accelerator and reactor neutrino oscillation searches (horizontal, dotted contours). Pure 
z^/i ^ atmospheric v oscillations (right side of each panel) are excluded by accelerator and 
reactor data for > 2 x 10~^ eV^. There are no significant limits below ~ 10~^ eV^ from present 
accelerator and reactor searches. 

FIG. 12. Kamiokande distribution of multi-GeV electrons and muons as a function of 
the zenith angle 0, in absence of neutrino oscillations. The agreement between the published 
Kamiokande simulation (solid histogram) and our calculation (dashed histogram) is very good. 
See Appendix A for details. 
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V mass spectrum 

scenario (a) scenario (b) 
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FIG. 1. The neutrino mass spectrum adopted in this work. One neutrino mass eigenstate (^3) 
is assumed to be largely separated from the quasi-degenerate doublet (i/i, v^) by a square mass 

difference jmg — 2 1 — The two possible scenarios (a) and (b) are physically different when 
the earth matter effects are included in the atmospheric f propagation. The two square mass 
spectra in (a) and (b) are related by: (a) (b) <^=^ — >■ —rm?. 
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i/^O i^^ oscillations 




FIG. 2. Analysis of separate and combined atmospheric neutrino data assuming pure ^ Vr 
oscillations (0 = in our framework) in the mass-mixing plane (tan^ -0, m?). Contours at 90% and 

99% C.L. for A^DF = 2 are shown. The allowed regions are marked by stars. The last panel shows 
the constraints coming from the combination of established accelerator data. Notice the symmetry 
of all the allowed regions with respect to the axis ip = ir/A. 
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tanV 

FIG. 3. Effects of the reduction of the neutrino flux uncertainty from the default value of ±30% 
(solid lines) to ±15% (dotted lines). The variations in the regions allowed at 90% C.L. by sub-GeV 
data (upper panel) or sub-GeV+multi-GeV data (lower panel) are very small. 
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FIG. 4. Separate and combined analysis of atmospheric neutrino data assuming pure ^ 

oscillations (^/^ = 7r/2 in our framework), in the mass-mixing plane (tan^ (?!>, m^). Contours at 90% 
(solid) and 99% C.L. (dotted) for A^df = 2 are shown. The allowed regions arc marked by stars. 
The last panel shows the constraints coming from the combination of established accelerator and 
reactor data. The contours of the atmospheric v allowed regions are not symmetric with respect 
to the axis ^ = 7r/4, due to matter oscillation effects. 
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FIG. 5. Bin-by-bin analysis of multi-GeV Kamiokande data in the plane of the // and e lepton 
rates, normalized to their theoretical values without oscillations, /xq and cq. Solid ellipses: theo- 
retical predictions at la level (A^^ = !)• Dotted ellipses: experimental data at la level. Notice 
how the theoretical ellipses change from the upper panel (no oscillation) to the three lower panels 
(best-fit cases for two-flavor and three-flavor oscillations). The fit is better in the Ug case 

than in the Vt case. The overall fit improves slightly in the oscillation case. 
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FIG. 6. As in Fig. 4, but excluding the earth matter effect (pure vacuum oscillations). All 
the contours are now symmetric with respect to the axis ^ = 7r/4. The regions allowed by the 
atmospheric neutrino data in the lower half of each panel are substantially different from those 
reported in Fig. 4. 
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Atmospheric sub— GeV & multi— GeV data 
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FIG. 7. Three-flavor analysis of all the atmospheric neutrino data (sub-GeV and binned multi- 

GeV combined) in the plane (tan^ ijj, tav? (J)), for 12 different values of rri^ ranging from 1.8 X lO'"''' 
to 3.2 X 10^'^ cV^. Scenario (a) of Fig. 1 is assumed. The solid (dotted) curves represent sections 
of the region allowed at 90% (99%) C.L. for A^df = 3 at given m^. The right side of each panel 
corresponds asymptotically to pure ^ oscillations; the lower side to pure ^ Ut oscillations. 
Three-flavor oscillations interpolate smoothly between these two limits. 
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Atmospheric sub— GeV 
& multi-GeV binned data 
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FIG. 8. Value of Ax^ for all atmospheric neutrino data (sub-GeV and multi-GeV combined) as 
a function of only. This figure embeds the information of Fig. 7 projected onto the rn? variable. 
At 68% C.L. (A^DF = 3) the value of m? is constrained between ~ 10^'^ and ^ 10^"*^ eV^. For very 
large m^, the value of Ax^ tends to ~ 7 (not shown) and there are no upper limits on at 95% 
C.L. The reduction of the v flux error from 30% to 20% does not produce significant variations, as 
indicated by the thin, dotted curve. 
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Atmospheric sub— GeV & multi— GeV data 
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FIG. 9. As in Fig. 7, but in the scenario (b) of Fig. 1. 
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Atmospheric sub— GeV 
& multi-GeV binned data 
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FIG. 10. As in Fig. 8, but in the scenario (b) of Fig. 1. 
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Atmospheric, Accelerator & Reactor data 
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FIG. 11. Comparison between the regions allowed at 90% C.L. (A^df = 3) by the atmospheric 
neutrino data in scenario (a) (solid contours), and the corresponding regions excluded by the 

established accelerator and reactor neutrino oscillation searches (horizontal, dotted contours). Pure 
J^/i ^ atmospheric v oscillations (right side of each panel) are excluded by accelerator and 
reactor data for m? ^ 2 x 10~^ eV^. There are no significant limits below ~ 10~^ eV^ from present 
accelerator and reactor searches. 
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FIG. 12. Kamiokande distribution of multi-GeV electrons and muons as a function of the zenith 
angle 0, in absence of neutrino oscillations. The agreement between the published Kamiokande 
simulation (solid histogram) and our calculation (dashed histogram) is very good. See Appendix A 
for details. 
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